6.7. Promotion and Rollback

Why gate a rollout on evaluation?

A healthy process can still be a worse agent.

A readiness probe proves the server starts. It cannot prove the source composition calls the right tools, stays grounded, or follows the evaluated operating contract. Those are release properties, so a build/deploy handoff needs behavior evidence first.

mise run promote packages that reasoning into a preflight:

flowchart LR
    Candidate["clean source commit"] --> Validate["offline eval-set validation"]
    Validate -->|fail| Stop(["stop"])
    Validate -->|pass| Model{"--with-model?"}
    Model -->|no| Render["render overlay"]
    Render --> Offline(["preflight only<br/>no deploy command"])
    Model -->|yes| Evals["trajectory + groundedness evals"]
    Evals -->|fail| Stop
    Evals -->|pass| Render
    Render --> Command["print commit-guarded<br/>Skaffold command"]
    Command -. "human runs later" .-> Build["build + deploy image"]
    Build --> Evidence["record digest<br/>scan + smoke same artifact"]

The script never builds or applies anything. It validates source inputs and manifests, then may print the next command for a human to review.

What does mise run promote prove?

The default command is an offline preflight, not permission to deploy.

mise run promote

It runs three ordered checks:

1. eval:validate verifies that committed eval cases and seed references agree.
2. The model-backed step is skipped and says how to enable it.
3. kubectl kustomize renders the selected overlay.

When those pass, the script exits zero but prints no deployment command. Offline validation proves the dataset and manifests are coherent; it has not measured candidate behavior.

Pass --with-model only when the configured model is ready:

mise run promote -- --with-model

Step 2 then runs eval:mlflow for deterministic conversation scorers and eval:ground for evidence grounding. This mode requires a clean Git worktree and records its commit. It explicitly clears AGENT_PROMPT_URI, so both evaluations use the committed instruction the production image will contain; use eval:ab separately for registry experiments. A failure or later source change stops before any command is emitted. Only a green model-backed run plus a clean render prints a commit-guarded skaffold run command.

The evaluations use your current model configuration; selecting an overlay does not switch their provider, model, or endpoint. Record those inputs and separately qualify the target model/tool path. Native Gemini evidence alone does not prove the gateway transport, and Ollama evidence does not qualify Vertex.

How does the target overlay change the command?

Select exactly one overlay: local-gemini by default, optional local for Ollama, or gke.

mise run promote -- local-gemini --with-model
mise run promote -- local --with-model
mise run promote -- gke --with-model

Each path renders infra/k8s/overlays/<overlay>. The repository in the emitted command differs:

Unknown flags, unknown overlays, and a second overlay fail fast. The GKE path still prints a command only; it does not run tofu apply, create a cluster, or deploy.

Both local commands require the k3d-local context before running Skaffold and preserve the selected profile. 6.2. Platform Install owns cluster and credential preparation; a printed command does not establish those prerequisites.

What must a real rollback record?

This preflight evaluates a clean source commit, not an image.

The serving image contains the committed INSTRUCTION; it deliberately omits MLflow. AGENT_PROMPT_URI is available only to host development and evaluation processes, so a prompt-registry URI is not a production rollback lever in this repository.

Use the prompt registry to compare wording before committing it. The printed Skaffold command then builds an image from that unchanged commit. Only after the build does an image digest exist, and the source evaluation alone does not prove that artifact starts, is clean, or contains the expected code.

For a production release, scan and smoke the exact immutable digest you will deploy, then record it with its source commit. If a release regresses, redeploy the previous known-good digest. This lab stops at the explicit build/deploy handoff rather than pretending to provide that image-provenance pipeline.

This course does not claim an automated canary. The local platform has one replica, no traffic-weighted stable/canary route, and no online scorer that can make a safe automatic rollback decision.

How do you prove a regression cannot reach promotion?

Use one reversible eval-set edit and no cluster.

1. Run mise run promote. Confirm all three steps pass, the overlay renders, and the final message says no promotion command was emitted.
2. In agents/python/evals/ops.evalset.json, find the incident-detail case and temporarily change its expected get_incident argument from INC-001 to absent id INC-042.
3. Run mise run promote again. Confirm it stops inside [1/3] before rendering and emits no deployment command.
4. Change INC-042 back to INC-001, inspect git diff, and run mise run promote once more.

INC-999 is not suitable for this drill: the eval set deliberately uses it as a negative case and the validator knows it should remain absent.

The optional final proof needs a model:

mise run doctor:model
mise run promote -- --with-model

Read the printed command; do not execute it for this page. Clean source behavior evidence makes a build/deploy command eligible for human review. It neither authorizes a rollout nor evaluates the image that command will build.

What proves this page worked?

The offline preflight and the deliberate regression are sufficient.

You are done when:

• mise run promote passes without a model and emits no deployment command.
• The temporary invalid eval reference stops inside [1/3], before the overlay render.
• The eval set is restored, git diff shows no accidental drill residue, and the offline preflight passes again.
• You can explain why only --with-model may emit a command and why the script still never applies it.
• You can name the handoff: evaluated source commit, then build, scan, smoke, deploy, and record one immutable image digest.

Continue to Observability when “preflight passed” and “safe to deploy” no longer mean the same thing.